On Using the Cloud to Support Online Courses

The increasing interest of online learning is unquestionable nowadays, with MOOCs being taken by thousands of students. However, for online learning to go mainstream it is necessary that professors perceive that the effort required to prepare and manage an online course is manageable. Today, a myriad of inexpensive tools and services can be used to produce and manage online courses with unprecedented ease and without distressing the professor. For that, this paper proposes an architecture based on Cloud services that simplifies the process of managing an online course, from delivering on-demand fully customized remote laboratories to communication automation for student engagement and feedback gathering. This approach has been applied to produce, distribute and manage an Online Course on Cloud Computing with Amazon Web Services. The paper describes the methodology, tools and results of this experience to point out that it is possible to deliver online courses with automatically provisioned labs, with minimal management overhead, while still providing a high quality learning experience to a worldwide audience.Moltó, G.; Caballer Fernández, M. (2014). On Using the Cloud to Support Online Courses. Frontiers in Education Conference. 2014:330-338. doi:10.1109/FIE.2014.7044041S330338201

WINGS: Worflow In Next generation Grids

[ES] WINGS (Workflow In Next Generation Grids) es un sistema workflow independiente de la plataforma diseñado para facilitar la creación de aplicaciones grid. Está compuesto por dos elementos: el lenguaje de definición WINGS y el entorno de ejecución WINGS (WINGS-RT). El primero es un lenguaje XML que utiliza conceptos similares de trabajos anteriores, pero añadiendo nuevos conceptos y evitando la creación de la definición para un entorno o aplicación en concreto. Así, uno de los principales objetivos del lenguaje WINGS es proporcionar la capacidad de ser evolucionado por los usuarios, añadiendo nuevos elementos con nuevas funcionalidades con el fin de adecuarlo a los middlewares actuales y futuros. El sistema de ejecución se encarga de proveer la funcionalidad definida en el lenguaje de definición XML, creando un entorno paralelo para el lanzamiento de los trabajos.[EN] WINGS (Workflow In Next Generation Grids) is a platform-agnostic workflow system designed to make the creation of grid applications easier. It is composed by two elements: the WINGS description language and the WINGS runtime environment (WINGS-RT). The first one is a XML language that uses similar concepts used in previous works, but adding new concepts and avoiding creating the definition for a concrete application or environment. Thus one of the main objectives of the WINGS language is provide the capability of being evolved by the users, adding new elements with new functionality in order to adequate to the current and future middlewares. The runtime system is in charge of providing the functionality defined in the XML description language, creating a parallel environment to launch the jobs.Caballer Fernández, M. (2012). WINGS: Worflow In Next generation Grids. http://hdl.handle.net/10251/19222Archivo delegad

Gestión de infraestructuras virtuales configuradas dinámicamente

En los últimos años y con el auge las tecnologías de virtualización y de las infraestructuras cloud, se abre un nuevo abanico de posibilidades para acceso de recursos de cómputo para el ámbito científico. Estas tecnologías permiten "acceso ubicuo, adaptado y bajo demanda en red a un conjunto compartido de recursos de computación". Estas tecnologías permiten que el acceso a grandes cantidades de recursos virtualizados sea mucho más sencillo para el científico. Si bien la adaptación de aplicaciones a un entorno distribuido sigue requiriendo de una experiencia importante, es posible utilizar de forma eficiente software adaptado a sistemas de colas e incluso computación paralela de memoria distribuida. A pesar de todo, en la actualidad existen diferentes proveedores cloud, diferente software para el despliegue de plataformas cloud, diferentes gestores de máquinas virtuales, y otros componentes que complican el acceso de forma sencilla y homogénea. Por tanto el objetivo principal de esta tesis es la de proporcionar a la comunidad científica el acceso a las tecnologías de virtualización y cloud de manera sencilla. De tal manera que sea muy sencillo el despliegue y gestión de sus infraestructuras virtuales, para que los investigadores solo tengan que centrarse en las tareas propias de su aplicación. Una plataforma Cloud para investigación debe contemplar todos los aspectos necesarios para creación y gestión de las infraestructuras, partiendo de que el investigador debe poder expresar sus requerimientos, tanto hardware como software, sobre los recursos que va a necesitar para la ejecución de su aplicación. En base a los requerimientos definidos por el usuario el sistema debe crear la infraestructura del usuario, teniendo en cuenta aspectos como la selección de despliegues cloud, de imágenes de máquinas virtuales, procesos de contextualización, etc. El sistema también debe permitir que el usuario modifique la cantidad de recursos (elasticidad horizontal) así como las características de los mismos (elasticidad vertical). Por último la plataforma debe proporcionar interfaces tanto a nivel de usuario, mediante aplicaciones de comandos o interfaces gráficas, como a nivel programático para que capas de mayor nivel puedan hacer uso de la funcionalidad mediante un API. La tesis pretende tanto avanzar en las especificaciones y arquitecturas software como desarrollar y testear un prototipo.Caballer Fernández, M. (2014). Gestión de infraestructuras virtuales configuradas dinámicamente [Tesis doctoral]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/37376TESISPremios Extraordinarios de tesis doctorale

A framework and a performance assessment for serverless MapReduce on AWS Lambda

[EN] MapReduce is one of the most widely used programming models for analysing large-scale datasets, i.e. Big Data. In recent years, serverless computing and, in particular, Functions as a Service (FaaS) has surged as an execution model in which no explicit management of servers (e.g. virtual machines) is performed by the user. Instead, the Cloud provider dynamically allocates resources to the function invocations and fine-grained billing is introduced depending on the execution time and allocated memory, as exemplified by AWS Lambda. In this article, a high-performant serverless architecture has been created to execute MapReduce jobs on AWS Lambda using Amazon S3 as the storage backend. In addition, a thorough assessment has been carried out to study the suitability of AWS Lambda as a platform for the execution of High Throughput Computing jobs. The results indicate that AWS Lambda provides a convenient computing platform for general-purpose applications that fit within the constraints of the service (15 min of maximum execution time, 3008 MB of RAM and 512 MB of disk space) but it exhibits an inhomogeneous performance behaviour that may jeopardise adoption for tightly coupled computing jobs.This study was supported by the program "Ayudas para la contratacion de personal investigador en formacion de caracter pre-doctoral, programa VALid-d" under grant number ACIF/2018/148 from the Conselleria d'Educacio of the Generalitat Valenciana, Spain. The authors would also like to thank the Spanish "Ministerio de Economia, Industria y Competitividad" for the project "BigCLOE" with reference number TIN2016-79951-R.Giménez-Alventosa, V.; Moltó, G.; Caballer Fernández, M. (2019). A framework and a performance assessment for serverless MapReduce on AWS Lambda. Future Generation Computer Systems. 97:259-274. https://doi.org/10.1016/j.future.2019.02.057S2592749

Serverless computing for container-based architectures

[EN] New architectural patterns (e.g. microservices), the massive adoption of Linux contain- ers (e.g. Docker containers), and improvements in key features of Cloud computing such as auto-scaling, have helped developers to decouple complex and monolithic sys- tems into smaller stateless services. In turn, Cloud providers have introduced serverless computing, where applications can be defined as a workflow of event-triggered functions. However, serverless services, such as AWS Lambda, impose serious restrictions for these applications (e.g. using a predefined set of programming languages or difficulting the installation and deployment of external libraries). This paper addresses such issues by introducing a framework and a methodology to create Serverless Container-aware AR- chitectures (SCAR). The SCAR framework can be used to create highly-parallel event- driven serverless applications that run on customized runtime environments defined as Docker images on top of AWS Lambda. This paper describes the architecture of SCAR together with the cache-based optimizations applied to minimize cost, exemplified on a massive image processing use case. The results show that, by means of SCAR, AWS Lambda becomes a convenient platform for High Throughput Computing, specially for highly-parallel bursty workloads of short stateless jobs.The authors would like to thank the Spanish "Ministerio de Economia, Industria y Competitividad" for the project "BigCLOE" under grant reference TIN2016-79951-R. The authors would also like to thank Jorge Gomes from LIP for the development of the udocker tool.Pérez-González, AM.; Moltó, G.; Caballer Fernández, M.; Calatrava Arroyo, A. (2018). Serverless computing for container-based architectures. Future Generation Computer Systems. 83:50-59. https://doi.org/10.1016/j.future.2018.01.022S50598

Study of the influence of the needle lift on the internal flow and cavitationphenomenon in diesel injector nozzles by CFD using RANS methods

It is well known that cavitation phenomenon in diesel injector nozzles has a strong influence on the internal flow during the injection process and spray development. However, its influence on the flow during needle opening and closing remains still unclear due to the huge difficulties related to performing experiments at partial needle lifts. In this paper, an extended computational study has been performed in a multi-hole nozzle modeling 10 different fixed needle lifts. The internal flow has been modeled with a continuum nozzle flow model that considers the cavitating flow as a homogeneous mixture of liquid and vapour. Due to high Reynolds numbers, turbulence effects have been taken into account by RANS methods using a RNG k e model. Firstly, the code has been validated against experimental data at full needle lift conditions in terms of mass flow, momentum flux and effective velocity, showing a fairly good agreement with experimental results. Once the code has been validated, it has been possible to study in depth the internal nozzle flow and its characteristics at the outlet at different partial needle lifts. Nevertheless, not only the main flow features have been explained, but also the cavitation appearance and the turbulence development, which present huge differences between the different needle lifts simulated.The authors wish to acknowledge the Generalitat Valenciana for the financial support through the project GVA PROMETEO CMT 2010 (reference code: GR001/2009/00167539).Salvador, FJ.; Martínez López, J.; Caballer Fernández, M.; Alfonso Laguna, CD. (2013). Study of the influence of the needle lift on the internal flow and cavitationphenomenon in diesel injector nozzles by CFD using RANS methods. Energy Conversion and Management. 66:246-256. https://doi.org/10.1016/j.enconman.2012.10.0112462566

Orchestrating Complex Application Architectures in Heterogeneous Clouds

[EN] Private cloud infrastructures are now widely deployed and adopted across technology industries and research institutions. Although cloud computing has emerged as a reality, it is now known that a single cloud provider cannot fully satisfy complex user requirements. This has resulted in a growing interest in developing hybrid cloud solutions that bind together distinct and heterogeneous cloud infrastructures. In this paper we describe the orchestration approach for heterogeneous clouds that has been implemented and used within the INDIGO-DataCloud project. This orchestration model uses existing open-source software like OpenStack and leverages the OASIS Topology and Specification for Cloud Applications (TOSCA) open standard as the modeling language. Our approach uses virtual machines and Docker containers in an homogeneous and transparent way providing consistent application deployment for the users. This approach is illustrated by means of two different use cases in different scientific communities, implemented using the INDIGO-DataCloud solutions.The authors want to acknowledge the support of the INDIGO-Datacloud (grant number 653549) project, funded by the European Commission's Horizon 2020 Framework Program.Caballer Fernández, M.; Zala, S.; López, Á.; Moltó, G.; Orviz, P.; Velten, M. (2018). Orchestrating Complex Application Architectures in Heterogeneous Clouds. Journal of Grid Computing. 16(1):3-18. https://doi.org/10.1007/s10723-017-9418-yS318161Aguilar Gómez, F., de Lucas, J.M., García, D., Monteoliva, A.: Hydrodynamics and water quality forecasting over a cloud computing environment: indigo-datacloud. In: EGU General Assembly Conference Abstracts, vol. 19, p 9684 (2017)de Alfonso, C., Caballer, M., Alvarruiz, F., Hernández, V.: An energy management system for cluster infrastructures. Comput. Electr. Eng. 39(8), 2579–2590 (2013). http://www.sciencedirect.com/science/article/pii/S0045790613001365Amazon Web Services (AWS): Amazon Web Services (AWS). https://aws.amazon.com/ (2017)Amazon Web Services (AWS): CloudFormation. https://aws.amazon.com/cloudformation/ (2017)Apache Software Foundation: Apache Mesos. http://mesos.apache.org/ (2017)ARIA, ARIA. http://ariatosca.incubator.apache.org/ (2017)Bumpus, W.: NIST Cloud Computing Standards Roadmap. Tech. rep., National Institute of Standards and Technology (NIST). https://doi.org/10.6028/NIST.SP.500-291r2 (2013)Caballer, M., Blanquer, I., Moltó, G., de Alfonso, C.: Dynamic management of virtual infrastructures. J Grid Comput. 13(1), 53–70 (2015). https://doi.org/10.1007/s10723-014-9296-5Campos Plasencia, I., Fernández-del Castillo, E., Heinemeyer, S., López García, Á., Pahlen, F., Borges, G.: Phenomenology tools on cloud infrastructures using OpenStack. Eur. Phys. J. C 73(4), 2375 (2013). https://doi.org/10.1140/epjc/s10052-013-2375-0Celar: Celar. http://www.cloudwatchhub.eu/celar (2017)Chen, Y., de Lucas, J.M., Aguilar, F., Fiore, S., Rossi, M., Ferrari, T.: Indigo: building a datacloud framework to support open science. In: EGU General Assembly Conference Abstracts, vol. 18, p 16610 (2016)Chronos: Chronos. https://mesos.github.io/chronos/ (2017)Cloudify: Cloudify. http://getcloudify.org (2017)Davidović, D., Cetinić, E., Skala, K.: European research area and digital humanitiesDistefano, S., Serazzi, G.: Performance driven WS orchestration and deployment in service oriented infrastructure. J Grid Comput. 12(2), 347–369 (2014). https://doi.org/10.1007/s10723-014-9293-8EGI FedCloud: EGI FedCloud. https://www.egi.eu/federation/egi-federated-cloud/ (2017)Eucalyptus: Eucalyptus. https://www.eucalyptus.com/ (2017)Fiore, S., D’Anca, A., Palazzo, C., Foster, I., Williams, D.N., Aloisio, G.: Ophidia: toward big data analytics for eScience. Procedia Comput. Sci. 18, 2376–2385 (2013). https://doi.org/10.1016/j.procs.2013.05.409Fiore, S., Palazzo, C., D’Anca, A., Elia, D., Londero, E., Knapic, C., Monna, S., Marcucci, N.M., Aguilar, F., Płóciennik, M., et al.: Big data analytics on large-scale scientific datasets in the indigo-datacloud project. In: Proceedings of the Computing Frontiers Conference, pp 343–348. ACM (2017)Fiore, S., Płóciennik, M., Doutriaux, C., Palazzo, C., Boutte, J., żok, T., Elia, D., Owsiak, M., D’Anca, A., Shaheen, Z., et al.: Distributed and cloud-based multi-model analytics experiments on large volumes of climate change data in the earth system grid federation eco-system. In: 2016 IEEE International Conference on Big Data (Big Data), pp 2911–2918. IEEE (2016)Galante, G., Erpen de Bona, L.C., Mury, A.R., Schulze, B., da Rosa Righi, R.: An analysis of public clouds elasticity in the execution of scientific applications: a survey. J Grid Comput.,1–24. https://doi.org/10.1007/s10723-016-9361-3 (2016)Google Cloud Platform (GCP): Google Cloud Platform (GCP). https://cloud.google.com/ (2017)Hochstein, L. (ed.): Ansible: Up and Running, Automating Configuration Management and Deployment the Easy Way. O’Reilly Media (2014)Idabc: European Interoperability Framework for pan-European eGovernment Services. European Commission version 1, 1–25. https://doi.org/10.1109/HICSS.2007.68 (2004)IM: IM. http://www.grycap.upv.es/im (2017)INDIGO-DataCloud: D1.8 - General Architecture. Tech. rep., INDIGO-DataCloud Consortium (2015)INDIGO-DataCloud: Ansible Galaxy repository for INDIGO-DataCloud. https://galaxy.ansible.com/indigo-dc/ (2017)INDIGO-DataCloud: Disvis/Powerfit Ansible Role in Ansible Galaxy. https://galaxy.ansible.com/indigo-dc/disvis-powerfit/ (2017)INDIGO-DataCloud: INDIGO-DataCloud. https://www.indigo-datacloud.eu/ (2017)INDIGO-DataCloud: INDIGO-DataCloud DockerHub application repository. https://hub.docker.com/u/indigodatacloudapps/ (2017)INDIGO-DataCloud: INDIGO-DataCloud PaaS Orchestrator. https://github.com/indigo-dc/orchestrator (2017)INDIGO-DataCloud: INDIGO-DataCloud RepoSync. https://github.com/indigo-dc/java-reposync (2017)INDIGO-DataCloud: INDIGO-DataCloud TOSCA templates. https://github.com/indigo-dc/tosca-templates (2017)INDIGO-DataCloud: TOSCA Across Clouds. https://github.com/indigo-dc/tosca-types/blob/master/examples/web_mysql_tosca_across_clouds.yaml (2017)INDIGO-DataCloud: TOSCA template for deploying an Elastic Mesos Cluster. http://github.com/indigo-dc/tosca-types/blob/master/examples/mesos_elastic_cluster.yaml (2017)INDIGO-DataCloud: TOSCA template for Powerfit application. https://github.com/indigo-dc/tosca-types/blob/master/examples/powerfit.yaml (2017)Kacsuk, P., Kecskemeti, G., Kertesz, A., Nemeth, Z., Kovȧcs, J., Farkas, Z.: Infrastructure aware scientific workflows and infrastructure aware workflow managers in science gateways. J Grid Comput., 641–654. https://doi.org/10.1007/s10723-016-9380-0 (2016)Korambath, P., Wang, J., Kumar, A., Hochstein, L., Schott, B., Graybill, R., Baldea, M., Davis, J.: Deploying kepler workflows as services on a cloud infrastructure for smart manufacturing. Procedia Comput. Sci. 29, 2254–2259 (2014)Koski, K., Hormia-Poutanen, K., Chatzopoulos, M., Legrė, Y., Day, B.: Position Paper: European Open Science Cloud for Research. Tech. Rep. october, EUDAT, LIBER, OpenAIRE, EGI, GĖANT Bari (2015)Krieger, M.T., Torreno, O., Trelles, O., Kranzlmüller, D.: Building an open source cloud environment with auto-scaling resources for executing bioinformatics and biomedical workflows. Futur. Gener. Comput. Syst. 67, 329–340 (2017). https://doi.org/10.1016/j.future.2016.02.008Kurkcuoglu Soner, Z., Bonvin, A.: Science in the clouds: virtualizing haddock powerfit and disvis using indigo-datacloud solutions (2016)Lipton, P.C.T., Moser, S.I., Palma, D.V., Spatzier, T.I.: Topology and Orchestration Specification for Cloud Applications. Tech. rep., OASIS Standard (2013)Liu, C., Mao, Y., Van der Merwe, J., Fernandez, M.: Cloud Resource Orchestration: a Data-Centric Approach. In: Proceedings of the Biennial Conference on Innovative Data Systems Research (CIDR), pp 1–8. Citeseer (2011)López García, Á., Fernández-del Castillo, E.: Analysis of scientific cloud computing requirements. In: Proceedings of the IBERGRID 2013 Conference, p 147 158 (2013)López García, Á., Fernández-del Castillo, E., Orviz Fernández, P.: Standards for enabling heterogeneous IaaS cloud federations. Comput. Standard Inter. 47, 19–23 (2016). https://doi.org/10.1016/j.csi.2016.02.002López García, Á., Zangrando, L., Sgaravatto, M., Llorens, V., Vallero, S., Zaccolo, V., Bagnasco, S., Taneja, S., Pra, S.D., Salomoni, D., Donvito G.: Improved cloud resource allocation: how INDIGO-datacloud is overcoming the current limitations in cloud schedulers. arXiv: 1707.06403 (2017)Lorido-Botran, T., Miguel-Alonso, J., Lozano, J.A.: A review of auto-scaling techniques for elastic applications in cloud environments. J Grid Comput. 12(4), 559–592 (2014). https://doi.org/10.1007/s10723-014-9314-7Marathon: Marathon. https://mesosphere.github.io/marathon/ (2017)Metsch, T., Edmonds, A.: Open Cloud Computing Interface-Infrastructure. Tech. rep., Open Grid Forum (2010)Metsch, T., Edmonds, A.: Open Cloud Computing Interface-RESTful HTTP Rendering. Tech. rep., Open Grid Forum (2011)Microsoft Azure: Microsoft Azure. https://azure.microsoft.com/ (2017)Moltó, G., Caballer, M., Pérez, A., Alfonso, D.C., Blanquer, I.: Coherent application delivery on hybrid distributed computing infrastructures of virtual machines and docker containers. In: 2017 25Th Euromicro International Conference on Parallel, Distributed and Network-Based Processing (PDP). https://doi.org/10.1109/PDP.2017.29 , pp 486–490 (2017)Monna, S., Marcucci, N.M., Marinaro, G., Fiore, S., D’Anca, A., Antonacci, M., Beranzoli, L., Favali, P.: An Emso data case study within the indigo-Dc project. In: EGU General Assembly Conference Abstracts, vol. 19, p 12493 (2017)Nyrén, R., Metsch, T., Edmonds, A., Papaspyrou, A.: Open Cloud Computing Interface–Core. Tech. rep., Open Grid Forum (2010)OASIS: Organization for the Advancement of Structured Information Standards (OASIS). https://www.oasis-open.org (2015)Open Telekom Cloud (OTC): Open Telekom Cloud (OTC). https://cloud.telekom.de/en/ (2017)OpenNebula: OneFlow. http://docs.opennebula.org/5.2/advanced_components/application_flow_and_auto-scaling/index.html (2017)OpenNebula Project: OpenNebula. https://www.opennebula.org (2017)OpenStack Foundation: Heat Orchestration Template (HOT) Guide. https://docs.openstack.org/heat/latest/template_guide/hot_guide.html (2017)OpenStack Foundation: OpenStack. https://www.openstack.org (2017)OpenStack Foundation: Openstack Heat. http://wiki.openstack.org/wiki/Heat (2017)OpenStack Foundation: OpenStack Heat Translator. https://github.com/openstack/heat-translator (2017)OpenStack Foundation: OpenStack heat-translator project contribution statistics. http://stackalytics.com/?release=all&metric=commits&module=heat-translator (2017)OpenStack Foundation: OpenStack Tacker. https://wiki.openstack.org/wiki/Tacker (2017)OpenStack Foundation: OpenStack tosca-parser project contribution statistics. http://stackalytics.com/?release=all&metric=commits&module=tosca-parser (2017)OpenStack Foundation: TOSCA Parser. https://github.com/openstack/tosca-parser (2017)OpenTOSCA: OpenTOSCA. http://www.opentosca.org/ (2017)Owsiak, M., Plociennik, M., Palak, B., Zok, T., Reux, C., Di Gallo, L., Kalupin, D., Johnson, T., Schneider, M.: Running simultaneous kepler sessions for the parallelization of parametric scans and optimization studies applied to complex workflows. J Comput. Sci. 20, 103–111 (2017)Palma, D., Rutkowski, M., Spatzier T.: TOSCA Simple Profile in YAML Version 1.1. Tech. rep., OASIS Standard. http://docs.oasis-open.org/tosca/TOSCA-Simple-Profile-YAML/v1.1/TOSCA-Simple-Profile-YAML-v1.1.html (2016)Petcu, D.: Consuming resources and services from multiple clouds: from terminology to cloudware support. J Grid Comput. 12(2), 321–345 (2014). https://doi.org/10.1007/s10723-013-9290-3Plóciennik, M., Fiore, S., Donvito, G., Owsiak, M., Fargetta, M., Barbera, R., Bruno, R., Giorgio, E., Williams, D.N., Aloisio, G.: Two-level dynamic workflow orchestration in the INDIGO DataCloud for large-scale, climate change data analytics experiments. Procedia Comput. Sci. 80, 722–733 (2016). https://doi.org/10.1016/j.procs.2016.05.359Płóciennik, M., Fiore, S., Donvito, G., Owsiak, M., Fargetta, M., Barbera, R., Bruno, R., Giorgio, E., Williams, D.N., Aloisio, G.: Two-level dynamic workflow orchestration in the indigo datacloud for large-scale, climate change data analytics experiments. Procedia Comput. Sci. 80, 722–733 (2016)Python: Python Package Index (PyPI). https://pypi.python.org/pypi (2017)Ramakrishnan, L., Jackson, K.R., Canon, S., Cholia, S., Shalf, J.: Defining future platform requirements for e-Science clouds. In: Proceedings of the 1st ACM Symposium on Cloud Computing - SoCC ’10. https://doi.org/10.1145/1807128.1807145 , p 101 (2010)Ramakrishnan, L., Zbiegel, P.T.T.T.: Magellan: experiences from a science cloud. In: Proceedings of the 2Nd International Workshop on Scientific Cloud Computing. http://dl.acm.org/citation.cfm?id=1996119 , pp 49–58 (2011)Salomoni, D., Campos, I., Gaido, L., Donvito, G., Antonacci, M., Fuhrman, P., Marco, J., Lopez-Garcia, A., Orviz, P., Blanquer, I., et al.: Indigo-datacloud: foundations and architectural description of a platform as a service oriented to scientific computing. arXiv: http://arXiv.org/abs/1603.09536 (2016)Sánchez-Expósito, S., Martín, P., Ruiz, J.E., Verdes-Montenegro, L., Garrido, J., Sirvent, R., Falcó, A.R., Badia, R.M., Lezzi, D.: Web services as building blocks for science gateways in astrophysics. J Grid Comput. 14(4), 673–685 (2016). https://doi.org/10.1007/s10723-016-9382-ySlipStream: SlipStream. http://sixsq.com/products/slipstream/ (2017)Stockton, D.B., Santamaria, F.: Automating NEURON simulation deployment in cloud resources. Neuroinformatics 15(1), 51–70 (2017). https://doi.org/10.1007/s12021-016-9315-8Teckelmann, R., Reich, C., Sulistio, A.: Mapping of Cloud Standards to the Taxonomy of Interoperability in Iaas. In: 2011 IEEE Third International Conference on Cloud Computing Technology and Science (Cloudcom), pp 522–526. IEEE (2011)Toor, S., Osmani, L., Eerola, P., Kraemer, O., Lindén, T., Tarkoma, S., White, J.: A scalable infrastructure for CMS data analysis based on OpenStack Cloud and Gluster file system. J Phys.: Conf. Ser. 513(6), 062,047 (2014). https://doi.org/10.1088/1742-6596/513/6/062047 . http://stacks.iop.org/1742-6596/513/i=6/a=062047?key=crossref.84033a04265ce343371c7f38064e7143UK Government Cabinet Office: Open Standards Principles. https://www.gov.uk/government/publications/open-standards-principles/open-standards-principles (2015)Yangui, S., Marshall, I.J., Laisne, J.P., Tata, S.: Compatibleone: the open source cloud broker. J Grid Comput. 12(1), 93–109 (2014)Zhao, Y., Li, Y., Raicu, I., Lu, S., Tian, W., Liu, H.: Enabling scalable scientific workflow management in the cloud. Futur. Gener. Comput. Syst. 46, 3–16 (2015). https://doi.org/10.1016/j.future.2014.10.023van Zundert, G., Trellet, M., Schaarschmidt, J., Kurkcuoglu, Z., David, M., Verlato, M., Rosato, A., Bonvin, A.: The DisVis and PowerFit web servers: explorative and integrative modeling of biomolecular complexes. J. Mol. Biol. 429(3), 399–407 (2013). http://www.sciencedirect.com/science/article/pii/S002228361630527

Towards migratable elastic virtual clusters on hybrid clouds

© 2015 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.This paper describes the research work in the context of the CLUVIEM project towards achieving migrat- able, self-managed virtual elastic clusters on hybrid Cloud infrastructures. These virtual clusters can span across on- premises and public Cloud infrastructures thus leveraging hybrid Cloud platforms. They are elastic since working nodes are automatically provisioned and relinquished to dynamically adapt the capacity of the virtual cluster (in terms of number of nodes) according to the current workload. They are self- managed since the elasticity rules are managed via the head node without requiring any external software entity for mon- itoring and deciding when to scale in and out. Finally, they are migratable since they consider both application migration, via application checkpointing, and infrastructure migration, by cloning infrastructures across multi-Clouds. These features introduce unprecedented flexibility for cost-effective cluster- based computing with minimal impact for cluster users. The paper summarises the current state of developments and future roads to achieve this vision.AC would like to thank the program “Ayudas para la contratacion de personal investigador en formaci ´ on´ de carcter predoctoral, programa VALi+d”, grant number ACIF/2013/003, from the Conselleria d’Educacio of the ´ Generalitat Valenciana. Also, the authors would like to thank the Spanish ”Ministerio de Econom´ıa y Competitividad” for the CLUVIEM project with reference TIN2013-44390-RCalatrava Arroyo, A.; Moltó, G.; Romero Alcalde, E.; Caballer Fernández, M.; Alfonso Laguna, CD. (2015). Towards migratable elastic virtual clusters on hybrid clouds. IEEE. https://doi.org/10.1109/CLOUD.2015.139

Managing Workflows on top of a Cloud Computing Orchestrator for using heterogeneous environments on e-Science

[EN] Scientific workflows (SWFs) are widely used to model processes in e-Science. SWFs are executed by means of workflow management systems (WMSs), which orchestrate the workload on top of computing infrastructures. The advent of cloud computing infrastructures has opened the door of using on-demand infrastructures to complement or even replace local infrastructures. However, new issues have arisen, such as the integration of hybrid resources or the compromise between infrastructure reutilisation and elasticity. In this article, we present an ad hoc solution for managing workflows exploiting the capabilities of cloud orchestrators to deploy resources on demand according to the workload and to combine heterogeneous cloud providers (such as on-premise clouds and public clouds) and traditional infrastructures (clusters) to minimise costs and response time. The work does not propose yet another WMS but demonstrates the benefits of the integration of cloud orchestration when running complex workflows. The article shows several configuration experiments from a realistic comparative genomics workflow called Orthosearch, to migrate memory-intensive workload to public infrastructures while keeping other blocks of the experiment running locally. The article computes running time and cost suggesting best practices.This paper wants to acknowledge the support of the EUBrazilCC project, funded by the European Commission (STREP 614048) and the Brazilian MCT/CNPq N. 13/2012, for the use of its infrastructure. The authors would like also to thank the Spanish 'Ministerio de Economia y Competitividad' for the project 'Clusters Virtuales Elasticos y Migrables sobre Infraestructuras Cloud Hibridas' with reference TIN2013-44390-R.Carrión Collado, AA.; Caballer Fernández, M.; Blanquer Espert, I.; Kotowski, N.; Jardim, R.; Dávila, AMR. (2017). Managing Workflows on top of a Cloud Computing Orchestrator for using heterogeneous environments on e-Science. International Journal of Web and Grid Services. 13(4):375-402. doi:10.1504/IJWGS.2017.10003225S37540213

A Self-managed Mesos Cluster for Data Analytics with QoS Guarantees

[EN] This article describes the development of an automated configuration of a software platform for Data Analytics that supports horizontal and vertical elasticity to guarantee meeting a specific deadline. It specifies all the components, software dependencies and configurations required to build up the cluster, and analyses the deployment times of different instances, as well as the horizontal and vertical elasticity. The approach followed builds up self-managed hybrid clusters that can deal with different workloads and network requirements. The article describes the structure of the recipes, points out to public repositories where the code is available and discusses the limitations of the approach as well as the results of several experiments.The work presented in this article has been partially funded by a research grant from the regional government of the Comunitat Valenciana (Spain), co-funded by the European Union ERDF funds (European Regional Development Fund) of the Comunitat Valenciana 2014-2020, with reference IDIFEDER/2018/032 (High-Performance Algorithms for the Modelling, Simulation and early Detection of diseases in Personalized Medicine). The authors would also like to thank the Spanish "Ministerio de Economia, Industria y Competitividad" for the project "BigCLOE" with reference number TIN2016-79951-R.López-Huguet, S.; Pérez-González, AM.; Calatrava Arroyo, A.; Alfonso Laguna, CD.; Caballer Fernández, M.; Moltó, G.; Blanquer Espert, I. (2019). A Self-managed Mesos Cluster for Data Analytics with QoS Guarantees. Future Generation Computer Systems. 96:449-461. https://doi.org/10.1016/j.future.2019.02.047S4494619